Multivariate Generalizability Analyses of Mixed-Format Advanced Placement Exams

When combining item types within assessments, it is important to consider the reliability of scores for each item type and the reliability of composite scores. This study investigated the effect of scoring procedures, section weights, and numbers of items per section on reliability, error variance, and conditional standard errors of measurement, using multivariate generalizability theory techniques. Results indicate that the multiple-choice scoring method (number correct vs. formula scoring) may impact multiple-choice section and composite score reliability, and result in different optimal section weights. Although the multiple-choice section contributes between 50 and 60 percent of the composite score points operationally, optimal weighting from a reliability or error variance perspective would increase the multiple-choice contribution to the composite to 80 or 90 percent. Increasing the number of free-response items often improves the reliability of the composite, but administration time limits the number of free-response items that can be administered. The results of this study are necessarily specific to the Advanced Placement (AP) Biology and AP World History Exams considered, but these data serve to illustrate the usefulness of generalizability theory for answering test development questions that arise with mixed-formats exams.